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Development of a Scale for the 
Student Assessment of Academic Advisors 

Abstract 

The academy has been inundated with calls to assess the services we provide to 
our students. Academic advising is one such service. Effective advising is critical to 
student progress and student retention. Effective advising can provide both personal and 
professional benefits to faculty advisors. However, assessment of advising is crucial to 
ensuring effectiveness. This study reports the development of an index for the student 
assessment of academic advisors. Psychometric analyses demonstrate that the instrument 
is both reliable and valid. The 15-item instrument thus offers a potentially valuable tool 
for improving advising and rewarding excellent faculty advisors. 
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Development of a Scale for the 
Student Assessment of Academic Advisors 

Academic advising is important to students, to faculty and to the academy. The 
National Academic Advising Association (NACADA, 2000) claimed, "Few experiences 
in students' postsecondary career have as much potential for influencing their 
development as does academic advising." Wiseman and Sanders (1988) noted that 
faculty received substantial psychic benefits from advising relationships. Crockett (1985, 
p. 244) indicated, "Academic advising, effectively delivered, can be a powerful influence 
on student development and learning and as such, can be a potent retention force on 
campus." Clearly, the potential benefits of effective academic advising are significant. 
While the value of advising has been recognized for over a century in this country (Trent, 
1997) and its specific effect on retention recognized for at least two decades (Beal & 

Noel, 1 980), we currently find ourselves faced with the need to more effectively assess 
our academic advisors and advising systems. 

Assessment is the buzzword in the academy. As the Educational Resources 
Institute Clearinghouse commented in 1999, 

Although assessment has been a major trend in higher education for the last ten 
years, the nature of the literature has changed significantly over the years. For 
example, in the late 1980s, the literature urgently called attention to the need for 
colleges and universities to perform self-assessment and to be held accountable 
for the quality of teaching, student learning, and services on their campuses. In the 
early 1990s, the discussion turned to specific assessment techniques and 
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approaches. Assessment was often marginalized to peripheral services or to 
traditional indicators such as student graduation or employment rates. In the 
middle of this decade, assessment of core processes — teaching and learning — 
became part of the major higher education debates between practitioners and 
researchers. The discussion of assessment has now become institution-wide — 
student affairs practitioners and faculty are as likely now as business officers to be 
involved in discussions about assessment. However, the need for assessment, 
different assessment techniques and approaches, and the benefit or impact of 
assessment are still being debated and people continue to search for ways to 
understand the process. "Performance indicators," "benchmarking," "total quality 
management," "outcomes," and "assessment models" are terms being learned, 
refined, and implemented in the daily life of academe. (ERIC, 1999) 

Academic advisement is among the many services of the university that have been added 
to the list of functions that must be assessed. 

In this paper we examine the potential benefits of academic advising, the need for 
student assessment of advising and develop an instrument for student assessment of 
advising. It is hoped our efforts will assist academic advisors and institutions in their 
efforts to assess advisement, and recognize effective advisors. We also hope to enhance 
the educational experiences of our students. 

Academic Advisement Benefits to Students and the Academy 

Academic advisement is highly valued in U.S. universities (Trent, 1997). It is 
widely touted as a highly effective retention tool (Frost, 1991). Why is advising so 
consequential? Advising influences a student's involvement in the university. Student 
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involvement in the university is a key factor in retention (Astin, 1984). Involvement 
highly influences student satisfaction and persistence (Astin, 1985). While a number of 
factors influence involvement, interaction with faculty is particularly significant (Tinto, 
1987). 

There has been considerable research on why students drop out of college. For 
example, Higgerson (1985), in her large-scale survey of students who had left college, 
found three major reasons for student attrition: (1) the student was dissatisfied with the 
academic program, (2) the student had unclear career objectives, and (3) the student had 
unclear educational goals. Academic and career uncertainty undermine students' 
motivations to continue their education. Support for this contention is inferred from 
Wessell, Engle, and Smidchens' (1978) study of college students which determined that 
students committed to a major had a significantly higher retention rate than those who were 
undecided. The Task Force on Academic Advisement of California State University, 
Northridge (1979) discovered that many students were in need of personal help in planning 
their educational and career goals. Habley (1981) noted that the advisor can be the key 
factor in helping students clarify goals and opportunities. It would seem that academic, 
social, and career integration are critical in a student's decision to stay in college. 

One of the best ways of insuring the students' understanding of and commitment to 
higher education is to enhance their academic and social integration into the campus. This 
requires systematic communication between institutional representatives and students. In 
their models of student retention, Spady (1970) and Tinto (1975) suggested that students 
who have extensive, high quality interactions with persons in the social system of the 
campus are more likely to continue their enrollment. As Vincent Tinto (1975, p. 96) 
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concluded, "Given individual characteristics, prior experiences and commitments ... it is 
the individual's integration into academic and social systems of the college that most 
directly relates to his continuance in college." In essence, students who do not feel a part of 
the campus are more likely to drop out. In their seminal study on student retention, Beal 
and Noel (1980) indicated that a caring environment will facilitate the students' academic 
and social integration in the campus. 

Student-faculty interaction is critical in creating a caring and supportive educational 
climate. Several studies have found that the interaction between faculty and students is one 
of the most important determinants of student retention (Pantages & Creedon, 1978; 
Pascarella & Terenzini, 1979; Ramist, 1981). It leads to both social and academic 
integration in the college for traditional students (Tinto, 1975) and for transfer students who 
disproportionately are minorities (Volkwein, King, & Terenzini, 1986). Student-faculty 
interaction is related to the students' academic performance (Pascarella & Terenzini, 1978), 
self-perceived intellectual growth (Pascarella & Terenzini, 1978, 1980; Wilson, Gaff, 
Dienst, Wood, & Bavry, 1975), and self-esteem (Astin, 1977). According to Alexander 
Astin (1977): 

Student-faculty interaction has a stronger relationship to student satisfaction with 
the college experience than any other involvement variable, or, indeed, any other 
student or institutional characteristic. Students who interact frequently with faculty 
are more satisfied with all aspects of their institutional experience, including student 
friendships, variety of courses, intellectual environment, and even administration of 
the institution, (p. 223) 
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Wilson et al. (1975) elaborated on this result when they report that students who interact 
more with faculty are sought out more by their peers, read more books for pleasure, are 
more likely to have graduate degree plans, and make more progress in specific academic 
skills. On the bases of these and other studies, we can conclude that faculty involvement 
with students has profound benefits for the students. In reviewing the role of advising in 
the process, Frost (1991) concluded, 

When considered together, conclusions about contact, involvement, and 
persistence have powerful implications for advising coordinators, for individual 
advisers, and for students. Advising is sometimes the only structured relationship 
that links students with concerned representatives of the institution. Within the 
boundaries of the relationship, it is possible to increase meaningful contact with 
faculty, help students become involved in their academic experiences, and 
encourage students to remain in college— desirable outcomes for students and for 
institutions. They suggest that the potential influence of academic advising is 
profound. (1991, p. 13) 

Academic Advisement Benefits to Faculty 

Clearly there are advising benefits to faculty as well. Advising offers faculty to 
form relationships with students beyond the classroom and receive the psychic benefits 
associated with being an effective helper (Wiseman & Sanders, 1988). Moreover, 
advising is often a component used in making decisions for retention, tenure and 
promotion (Nadler & Nadler, 1996; Sun, Valiga, & Gao, 1997). For example, 
consistent with the collective bargaining agreement of the California State University that 
recognizes advising students is a faculty member's professional responsibility (Article 
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20.1 b, Agreement between California Faculty Association and The Board of Trustees of 
the California State University Unit 3 - Faculty July 1, 1998 - June 30, 2001), some 
campuses have required that advising be assessed as an integral part of the retention, 
tenure and promotion agreement (e.g., University Manual of California State Polytechnic 
University, Pomona, 1999). 

However, even though advising may be in integral part of a faculty member's job 
and may require a significant expenditure of faculty time, crediting the faculty with such 
work is difficult because the primary nature of advising is oral communication. Even 
though such advising relationships can have lasting effects on students and faculty, they 
frequently do not result in the kind of documentation that lends itself to effective 
consideration in personnel decisions. That which cannot be documented is not rewarded 
(See, e.g., Nadler & Nadler, 1996 and citations therein). Frost (1991) noted that faculty 
most often identify the failure to reward advising as one of the components that most 
needs correction in current advising systems. Consequently, developing means to 
document and assess advising are clearly in the interest of the faculty involved. 

How Should Advising be Assessed? 

Ideally, the assessment of advising systems will contain many components. 
Among these components, student assessment of advisors is very important (Frost, 1991; 
NACADA, 2000; Sun et al., 1997). Such evaluation can help individual advisors 
improve their advising competence, can provide documentation of student outcomes, and 
can provide documentation necessary to reward effective faculty advisors (Frost, 1991). 

While not ignoring the need for other means of advising assessment, we see 
student assessments as central to the system. Consequently we sought to create an 
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instrument for such student assessment. Underlying the instrument was the idea that 
advising is a special form of communication relationship (Nadler & Nadler, 1996). 

"Good advisers are interested in establishing caring relationships with students" (Frost, 
1991, p. 64). Consistent with the literature on communication competence (e.g., 
Spitzberg & Bruner, 1991), advising communication must involve knowledge, 
motivation, and skills. While students cannot actually speak to and advisor's motivation, 
they can assess the nature of the relationship that is established in the advising situation. 
Frost (1991) suggested that students prefer personalized advising centered around 
academics (not personal concerns). Moreover, students are competent to make some 
knowledge assessments about their adviser's facility with university policies and 
regulations. Frost (1991) indicated that students prefer advisors to supply information 
about prescriptive items such as graduation and major requirements and to address 
broader concerns such as life goals and career planning. She specified that advisors must 
know academic policies and regulations. Finally, students can assess the communicative 
skill with which the advisor relationship is enacted in their specific instances. 

Methods 

Respondents 

A total of 197 undergraduate students from two large western universities 
volunteered to participate in a survey focusing on their advisors' skills and competencies. 
Of these respondents, 80.2% were communication majors, 75.1% had at least three terms 
of study at their respective university, and 62.9% were female. The average age of the 
respondents was 23.8 (s.d. = 6.3). Nearly one-fourth of the respondents (27.8%) were 
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the first in their families to attend college. Finally, over one-half of the respondents 
(52.6%) had changed their major at least once since matriculating into college. 
Questionnaire 

Based upon the review of the literature on advisor competencies, behaviors, and 
effects (e.g., Crockett, 985; Fago, 1997; Frost, 1991; Wiseman & Sanders, 1988), scales 
were generated to enable the assessment of advisor communication and advisement 
outcomes. In terms of advisor communication, 33 items were devised. The 33 items 
covered a wide range of communication behaviors, including advisor patience, 
approachability, empathy, eye contact, display of interest, use of voice, and willingness to 
share. Also, the items examined advisor technical skills, including the advisor's 
knowledge, career insights, understanding of university policy and procedures, personal 
experiences, and currency in the field and academia. Finally, the items assessed more 
affective dimensions of advisement, such as, ability to motivate, being inspirational, 
friendliness, and immediacy. 

A pilot study was undertaken to explore the adequacy of the 33 items that were 
extrapolated from the literature. A total of 57 undergraduate student participated in the 
pilot study. Based upon both a qualitative analysis of the respondents' open-ended 
statements and a quantitative analysis of their ratings of their advisors using the 33 items, 
some refinements were made to the items measuring advisor communication and 
competencies. More specifically, most of the negatively worded items (i.e., "not") were 
positively worded, some items were deleted due to a lack of clarity or lack of internal 
reliability, and a few more global items (e.g., "I like my advisor") were added. After 
these refinements were made, a total of 27 revised items were constructed. These items 
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are presented in Table 1. All of the items were rated on a five-point Likert scale, where 1 
= Strongly Disagree to 5 = Strongly Agree. 

A scale to assess the effectiveness of advisement was also developed. Based upon 
the advisor outcomes literature (cf. Astin, 1977; Beal & Noel, 1979; Frost, 1991; Tinto, 
1975), a total of nine items were formulated focusing on potential changes in the advisee 
as a result of the advisor's intervention. More specifically, the items assess advisees' 
assessments of their integration into the campus, knowledge of the major or career, 
feelings toward the major/career, confidence levels, interaction with other faculty, and 
potential for success. These nine items are presented in Table 2. All of the items were 
rated on a five-point Likert scale, where 1 = Strongly Disagree to 5 = Strongly Agree. 

The inter-item reliability was quite high for these nine items (a = .95), suggesting 
unidimensionality among the items. 

Data Analysis 

Methodologically speaking, the goal of the study was to develop an instrument to 
assess advisor communication. It was felt that the instrument should have several 
desirable psychometric characteristics. More specifically, the instrument should (1) be 
unidimensional (i.e., the items should be measuring a similar phenomenon), (2) have an 
adequate degree of variability (i.e., the items should not be constants with floor or ceiling 
effects on their variability), (3) have an adequate degree of predictive validity (i.e., the 
items should be correlated with positive outcomes presumed to be obtained from 
effective advisement), and (4) be parsimonious (i.e., the number of items should not be 
too cumbersome to prevent the efficient use of the scale). The results report the degree to 
which to instrument we have developed meet the above criteria, with the ultimate goal of 
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finding a scale to measure advisor behavior that is psychometrically sound and 
parsimonious. 

Results 

Reliability Analysis 

The results of the inter-item reliability analysis are presented in Table 1. The 
resultant standardized alpha for the 27 items was quite high (a = .97), suggesting a 
unidimensional character to the 27 items. As a further test to the dimensionality of the 27 
items, they were examined using a principle components analysis. Based upon the 
empirical criterion of the scree test, a two-factor solution was extracted. These two 
factors were then rotated using a Varimax criterion and the rotated factor solution was 
then assessed. A number of problems were detected in the two-factor solution: (1) the 
factor solution failed Thurstone's criterion for a simple structure, viz., that items should 
load only on one factor, (2) most of the items loaded only on the first factor, and (3) the 
two factors were very difficult to interpret and thus label. It was decided that a single 
dimension characterized the 27 items. 

The results presented in Table 1 make it possible to enhance the scale's parsimony 
by indicating possible items that could be eliminated based on their psychometric 
properties. More specifically, six items (4, 6, 8, 12, 14, and 22) were eliminated due to a 
low item-total correlation (< .70) or a low multiple correlation (R 2 < .60), suggesting 
greater degrees of unique variance in these items (as opposed to common or shared 
variance, which would be expected in unidimensional measures). 

While it is not a rigorous assessment of test-retest reliability, we did attempt to 
determine the measures' stability by repeating one of the items. The repeated item, "My 
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advisor helps me find out what college has to offer," was placed earlier in the battery of 
items (#10) and later in battery (#27). It was hoped that each respondent would evaluate 
his/her advisor the same way on the two items. The results verified this assumption: the 
correlation between the two items was .86 and the overall match of values used was 
81%). 

Item Variability 

Another psychometric property assessed was the variability in each item. For an 
item to have any discriminability, there needed to be an adequate degree of variance in 
the item. If, for example, there were ceiling effects due to a social desirability bias, the 
item would lose much of its discriminability. An examination of the results presented in 
Table 1 indicated that three items could be eliminated on the basis of this criterion: #11 
("My advisor treats me with respect"), #15 ("My advisor is well educated and very 
knowledgeable in the field"), and #19 ("My advisor possesses a high level of knowledge 
about my major"). 

Item Redundancy 

Another psychometric property that was assessed in order to enhance the scale's 
parsimony was the degree of redundancy among the items. It was felt that if two items 
were highly redundant, one of the items could be eliminated because its statistical 
information was being measured via another item. An examination of the correlation 
matrix for the items revealed only one pair of items with a high degree of redundancy (r > 
.85), namely, items #23 ("I feel comfortable with my advisor") and #27 ("I trust the 
information my advisor gives me"). Of the two items, the former had better 
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psychometric qualities on the other criteria being assessed (variability and inter-item 
reliability); thus, item #27 was eliminated. 

Predictive Validity 

A third psychometric property assessed was the ability of items to predict positive 
outcomes from the advisement. The fourth column of the results presented in Table 1 
(viz., the item-outcome correlation) provides information to make this assessment 
possible. On the basis of this criterion (r < .60), two more items were eliminated: #13 
(My advisor uses appropriate eye contact") and #17 ("My advisor possesses a high level 
of knowledge about the job"). 

Summary 

Based upon our analyses of the psychometric properties of the items, a 15 -item 
instrument was extracted from the original 27 items. The inter-item reliability for these 
15 items was very high (a =.97). The final 15 items extracted for Advisement Scale are 
presented in Table 3. The average for the mean summed scale was 3.74 (s.d. = .86), with 
a range of 1 .13 to 5.00. Further, the correlation between the Advisement Scale and the 
Outcomes Scale was .79 (p < .0001). 

Discussion and Conclusion 

The psychometric analyses suggest that the advisor assessment index developed 
here offers promise as a means of evaluating advisors. The instrument appears to be both 
valid and reliable. Moreover it is parsimonious making for ease of use. The nature of the 
instrument is such that individual items can be analyzed separately or can be summed to 
form an index score. 
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While the literature on communication competence might suggest that advisor 
assessment would require a multidimensional instrument, we did not find multiple 
dimensions here. This is consistent with other literature indicating that students make 
global evaluations of advisors (Fago, 1997). Moreover, it offers the benefit of creating an 
index instrument for which individual items can be summed and an overall summary 
score reported. 

The instrument also appears to be a valid assessment of advisors. The instrument 
is highly reliable yet individual items contain sufficient variability to suggest that a range 
of scores is possible. The items initially generated have a high degree of face validity. 
Further, the comparison with expected advisor outcomes measures suggests that the 
instrument effectively predicts the anticipated results of advisement. 

The advisor assessment index is also a compact means of assessing advisors. 

With only fifteen items, the time and difficulty of administration are small. Chances of 
respondent fatigue are low. Consequently, the instrument represents an efficient means 
for evaluating advisors. Such efficient evaluation can create the groundwork for both 
improving advising and rewarding excellent advisors. 

Clearly academic advising is important. Effective advising can help students 
shape their academic and professional careers. Effective advising can contribute to 
student retention. Effective advising can also be a source of reward to faculty advisors 
both personally and professionally. The ability to assess advisors is critical to creating 
effective advising (Frost, 1991). While student assessment is not the only means for such 
evaluations, it is clearly an important one because students are the clients served in the 
advisement process. The advisor assessment index developed here provides a reliable, 
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parsimonious, and valid means for such assessment, and, in so doing, helps to meet the 
call for the academy to assess the ways we provide services to our students. 
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Tablet. Advisor Communication Items 



Item- Item- 







Mean 


s.d. 


Total r 


El 


Outcome r 


1 . 


When I visit my A , he/she is helpful. 


4.05 


0.88 


.79 


.73 


.68 


2. 


My A is approachable when I want to speak to him or her. 


3.89 


1.07 


.81 


.77 


.64 


3. 


When I visit my A , he/she is understanding and patient. 


4.03 


0.92 


.82 


.76 


.61 


4. 


My A is available during his/her designated office hours. 


3.91 


0.96 


.64 


.56 


.48 


5. 


My A shows interest in me as a person. 


3.74 


1.11 


.83 


.75 


.63 


6. 


My A does not show interest in my academic progress. (R) 2.19 


0.94 


.46 


.32 


.40 


7. 


My A shows interest in my professional career. 


3.53 


0.95 


.77 


.72 


.70 


8. 


My A is able to access the information (online and offline) 3.48 


0.96 


.66 


.53 


.56 




necessary to advise me effectively. 












9. 


My A is genuinely interested in helping me succeed. 


3.77 


1.07 


.81 


.76 


.69 


10. 


My A helps me find out what college has to offer. 


3.70 


1.04 


.71 


.63 


.67 


11. 


My A treats me with respect. 


4.18 


0.82 


.78 


.73 


.59 


12. 


My A uses personal examples or talks about experiences 


3.51 


1.14 


.62 


.54 


.54 




she/he has had outside the educational institution. 












13. 


My A uses appropriate eye contact. 


4.09 


0.85 


.76 


.71 


.58 


14. 


I ask my A for advice on projects I am working on. 


3.02 


1.09 


.60 


.53 


.57 


15. 


My A is well educated and very knowledgeable in field. 


4.17 


0.80 


.71 


.72 


.53 


16. 


My A is an excellent advisor. 


3.70 


1.04 


.90 


.85 


.72 


17. 


My A possesses a high level of knowledge about job. 


4.07 


0.87 


.76 


.79 


.54 


18. 


My A possesses a high level of knowledge about 


3.75 


0.94 


.73 


.67 


.60 




the university and its regulations. 












19. 


My A possesses a high level of knowledge about major. 


4.10 


0.84 


.72 


.64 


.58 


20. 


My A has excellent communication skills. 


4.07 


0.96 


.83 


.78 


.62 


21. 


My adviser is motivational and inspirational. 


3.59 


1.08 


.85 


.78 


.70 


22. 


My A and I talk on a regular basis. 


2.62 


1.29 


.63 


.54 


.61 


23. 


I feel comfortable with my adviser. 


3.74 


1.16 


.87 


.84 


.67 


24. 


My A is approachable on any question or issue that needs 


3.70 


1.10 


.84 


.78 


.71 




to be discussed. 












25. 


My A is willing to spend extra time on situations that 


3.46 


1.07 


.77 


.65 


.65 




are complex. 












26. 


I like my A. 


3.92 


1.06 


.87 


.85 


.69 


27. 


I trust the information my A gives me. 


3.98 


0.99 


.79 


.77 


.66 
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Table 2 

Advisor Effects Items 



Mean s.d. 

1 . Because of my advisor, I feel more a part of the campus. 2.90 0.99 

2. As a result of my advisement, I am more knowledgeable about the requirements 3.69 1 .06 

of the major. 

3. Because of my advisor, I better understand the career possibilities for my major. 3.32 1 .06 

4. As a result of my advisement, I am more satisfied with the university. 3.19 1 .02 

5. As a result of contact with my advisor, I am more enthusiastic about my major. 3.29 1 .07 

6. Due to the advice I was given, I have gained more confidence in my abilities. 3.33 1 .03 

7. As a result of the advisement I received, I feel I will be more successful in school. 3.39 1 .01 

8. Because of the contact with my advisor, I will be better able to interact with 3.22 0.96 

other faculty. 

9. Because of my advisor, I am more motivated to complete my degree. 3.36 1 .07 
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Sr 



* 



Table 3 

Advisement Scale 

1. When I visit my advisor, he/she is helpful. 

2. My advisor is approachable when I want to speak to him or her. 

3. When I visit my advisor, he/she is understanding and patient. 

4. My advisor shows interest in me as a person. 

5. My advisor shows interest in my professional career. 

6. My advisor is genuinely interested in helping me succeed. 

7. My advisor helps me find out what college has to offer. 

8. My advisor is an excellent advisor. 

9. My advisor possesses a high level of knowledge about the university and 

its regulations. 

10. My advisor has excellent communication skills. 

1 1 . My advisor is motivational and inspirational. 

12. 1 feel comfortable with my advisor. 

13. My advisor is approachable on any question or issue that needs 

to be discussed. 

14. My advisor is willing to spend extra time on situations that 

are complex. 

15. I like my advisor. 
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